Domino Start Script Diagnostic Improvements

Some of the following additions made it already into an earlier version.
I actually started after Engage. During Engage I had a server hang and had to run, download and analyze NSDs from the hotel room.
This did lead to the first diagnostic additions.

• NSDs are now at your fingertips. You can open the latest NSD and see when it was created.
• If you have nshmailx installed, you can send NSDs directly by mail (independent from Domino with the small helper tool via SMTPS).
• Today I added disk space information reading the translog, DAOS, FT and NIF settings from notes.ini. This was a missing part, which needed some thought which disks to show.

Domino process explorer

But on top of this I added a Domino process explorer to more interactively look into Domino processes, without creating a NSD.
I added this mainly for my own diagnostic and performance troubleshooting.
You can interactively pick processes, dump processes or just single threads in fast sequence -- including auto refresh from 1-9 seconds.
In addition you can toggle on/off interactively logging those stacks to a file.
It's using the GNU debugger (gdb) in a similar way NSD uses it.

New interactive navigation model

The interactive selection of processes is a new menu navigation model I am trying out.
This includes cursor up/down, selecting via "enter" and ESC to go back
It needs some tricks to put the session into raw mode for the time of the menu operations to read the extended keys (up/down cursor and ESC).

There are more details to discover. I am still working on details.
I have submitted the changes to the Domino Start Script and the Domino Container project in parallel.
This means the new script and the enhancements are already available in the container environment as well if you switch to the development branch.
---
Some of the changes are really targeted to the very senior admins to analyze problems.
But over all the diagnostic enhancements are benefiting everyone.
I added some of this for my own needs (helping customers to troubleshoot) and there is more coming.
What I would really like to do is semaphore annotation.
But this needs changes either in Domino, or I would need to provide an add-on annotation tool to annotate the semaphore time.

Let me know if this is helpful and what might be missing.

• Comments [0]

Leveraging ZFS for Domino native or via Proxmox LXC containers

ZFS is a very interesting file system for many reasons.
It offers compression, deduplication, snapshots, encryption and a very flexible volume manager.
ZFS is also the file system leveraged by Proxmox as the strategic file-system for local disks.
On Proxmox you can use ZFS in three different ways

1. Proxmox host level
2. LXC container as a direct mount without another file system in the LXC container
3. VM with a zvol which is kind of a raw device provided to the VM to add it's own file-system on top

The direct mount into the LXC container is a very interesting option which I tested before.
But now bringing ZFS to the picture this might make even more sense.

Of course this option only makes sense if you use ZFS native on Proxmox.
If you are running a larger Proxmox cluster, your storage is likely to use other options like Ceph.
But the following is also intended as food for thought to look into your own optimized storage.

One way that always works is to provide ZFS storage to a server over NFS.
NFS support is part of ZFS and allows to access ZFS over a network.

A simple configuration could look like the following.
This scenario would work with any machine which supports native ZFS.
It could be a Linux machine or a Proxmox host. Or an appliance like TrueNAS.


-- NFS Server --

Server Side on Ubuntu

Install packages for ZFS and NFS

apt install zfsutils-linux nfs-kernel-server

Create a pool and a volume with the right attributes for backup

zpool create tank /dev/sdb
zfs create -o mountpoint=/local/backup tank/backup
zfs set atime=off tank/backup
zfs set dedup=on tank/backup
zfs set recordsize=16K tank/backup


Enable NFS read/write sharing for the volume

zfs set sharenfs="rw=@192.168.96.42/32" tank/backup


Client Side on Ubuntu

Install package for NFS client

apt install nfs-common

Create a directory and mount the NFS volume (leaving out special attributes like noatime etc )

mkdir -p /local/backup
chown notes:notes /local/backup
mount -t nfs 192.168.96.42:/local/backup /local/backup


The resulting performance for a Domino backup of larger NSF files:

Data Rate: 521.2 MB/sec


-- Proxmox LXC container mount --

If you are running a LXC container on Proxmox, you can create a ZFS volume and directly mount it into the LXC container without any additional overhead.

Create a mount with the right options

zfs create rpool/backup
zfs set atime=off rpool/backup
zfs set compression=lz4 rpool/backup
zfs set dedup=on rpool/backup
zfs set recordsize=16K rpool/backup
chown 101000:101000 /rpool/backup

Modify the settings of your LXC container for example /etc/pve/lxc/101.conf
Append the following type of line and restart your LXC container

lxc.mount.entry = /rpool/backup local/backup none bind,create=dir 0 0

Aligning the recordsize to 16K improves deduplication but reduces the performance a bit.
The performance is still almost double then using native NFS and the 16K block size is probably the better match.

Data Rate:   955.1 MB/sec  (with  16K recordsize)
Data Rate: 1,430.3 MB/sec  (with 128K recordsize)


Why is the performance better mounting a volume into the LXC container?

Using NFS the network connection is used. Even this is the local network on the same Proxmox host, this causes overheard and limits the performance to the speed of the network.
Leveraging the underlying ZFS directly does not have any overhead and provides the performance of the underlaying storage.

The fast SSDs used could provide much higher performance without de-duplication.
But this is de-duplicating ZFS write performance, which is quite impressive.

I have been using my mail file as test data. But in real life with more data the performance might drop. But this shows the potential of the setup.


Other benefits

Another big advantages using native ZFS volumes is the very flexible storage allocation.
Mounting a volume into the container allows you to use the full flexibility in contrast as you can see in below example.

The more I look into Proxmox and LXC containers the more I would want to conside LXC container on Proxmox for hosting Domino servers.

---

Example list of volumes:

root@pve:/rpool# zfs list
NAME                           USED  AVAIL  REFER  MOUNTPOINT
rpool                          482G  1.33T   104K  /rpool
rpool/ROOT                    4.46G  1.33T    96K  /rpool/ROOT
rpool/backup                  58.6G  1.33T  58.6G  /rpool/backup
rpool/data/subvol-100-disk-0  13.1G  86.9G  13.1G  /rpool/data/subvol-100-disk-0


Deduplication status of the pool after a couple of backups:

zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
rpool  1.81T   441G  1.38T        -         -     3%    23%  3.29x    ONLINE  -

• Comments [0]

A simple but very useful Domino OTS enhancement in 14.x

Now that Domino 14.0 is out for a while and we have Domino 14.5 available it is time to look into adding new functionality to my OTS scripts.

It's a challenge to work with new functionality if you also want to support back-level versions.
But this enhancement is helpful and I will start using it for additional server setups in context of the Domino Start Script and container project.

For additional server setups a fixed case-sensitive lookup can be problematic for example when finding the server document.

The trick is to not use the server name typed in by the admin for lookups starting with Domino 14.0 as shown below.
But in general a formula is the much more flexible way to search for an existing document.

My next wishlist item would be to be able to specify a formula on a document before compute with form is executed.
This would allow the same type of flexibility for updating document information.

I have just created an idea a minute ago --> https://domino-ideas.hcltechsw.com/ideas/DOMINO-I-3042.
Please vote if you find it useful.


Domino 12.x

    "documents": [
          {
            "action": "update",
            "findDocument": {"Type": "Server", "ServerName": "CN=my-domino-server/O=Acme" },
            ...

Domino 14.x

    "documents": [
          {
            "action": "update",
            "findDocument": "Type = {Server} & ServerName = @Username",
            ...

• Comments [0]

Traveler is a continuos delivery product with one code stream

Traveler was always a single code stream going forward in builds and labeling it with a version.
The team also does awesome work to have new Traveler versions work with any support Domino versions.

You only need to make sure you reinstall Traveler if you switch to a new major release, because the Traveler installer will take care of installing the right binaries for the underlaying Domino version.

What is new that HCL is removing older downloads from MHS. Today you can only download Traveler 14.5.

But product management is thinking about brining back at least one back level version. Which would be currently Traveler 14.0 FP2.

In general this is a good and important practice. But you have to be careful if you are running add-on applications or special device types like SecurePIM.
There has been change in the way Apple APNS push is implemented. Which should just for out of the box for normal devices.

https://help.hcl-software.com/traveler/14.5.0/whats_new_14.5.0.html

Token-based authentication for APNS

-- snip--
Authentication by Traveler to the APNS servers has changed from certificate-based authentication to token-based authentication.
This removes the need to have or maintain the p12 certificate files for APNS which expire and therefore must be updated every year.
Customers using Maas360 and SecurePIM clients should check with the app providers for support of Traveler 14.5 APNS push support.
-- snip--

Updating Traveler

But beside that you are on the safe side updating to Traveler 14.5 for all supported Domino versions.
Below is the support statement from HCL, which is a bit hidden in a technote.

To avoid more questions I am writing this blog post to raise awareness.
There are also other Domino+ products like Verse, Nomad and the REST API which are on the continuos delivery model.

Domino Container project

The container project always selects the latest Traveler version and also other continuos automatically.

I am going to remove the very old software releases from software.txt in the Domino Container project.
But on purpose leaving some older web-kits in case admins have already downloaded the software earlier and need those older versions for example for testing upgrades etc.


HCL Traveler support statement

https://support.hcl-software.com/csm?id=kb_article&sysparm_article=KB0036438


-- snip--
There is one single maintenance stream being supported. This single release can be installed and run on any supported level of Domino server.
To obtain the latest features and customer fixes it is required to be at this latest Traveler level.
The one release model is necessary to keep pace with an extremely fast moving mobile marketplace and will allow all customers to stay current with the latest features and fixes for mobile devices.
-- snip--

• Comments [0]

Domino Container Environment Install Script

It has been a round for a while. I am too lazy to do the same installations over and over again manually.
In addition it helps me to get customer environments all setup in the same way.

This script clones the Domino container project and Domino start script.
It also installs Docker from the official repository along with the Domino download script and also the container control script.
This is the fastest way to setup a container environment.

I have just added a documentation page for the script:

https://nashcom.github.io/domino-startscript/install_container_env/

• Comments [0]

DACHNUG Vienna DNUG Lab preparation done

There are many interesting new features in Domino 14.5.
And I am working on a couple projects.


Beside updating the DNUG Lab to Domino 14.5, I also setup a lot of components on my demo notebook.

• Notebook with Windows 11 (https://blog.nashcom.de/nashcomblog.nsf/dx/black-friday-getting-some-new-ai-ready-hardware.htm)
• NVIDIA GeForce RTX GPU with 8 GB RAM.
• Latest Docker Desktop
   
• The NVIDIA GPU is running native on Windows, on Docker Desktop and also in WSL and inside Docker containers running on WSL Linux like Ubuntu.
• WSL with Ubuntu 24.04, Alma Linux 10, Kali Linux
• Docker container on Linux in different modes (Docker Desktop, native Linux in WSL, ...)
   
• The new Docker model runner
• VMware Workstation 17 Pro with Windows 2025
• NGINX on Linux and Windows as a host for the Domino IQ workshop on Monday

There is a lot to discover and we can show a lot of new functionality.

• Comments [0]

Let’s Encrypt stops OCSP support

Let's Encrypt is stopping OCSP support and goes back to a classical CRL.
You find details about this change here -> https://letsencrypt.org/2024/12/05/ending-ocsp/

I blogged about OCSP stampling a while ago. And just updated the post with this new information.

https://blog.nashcom.de/nashcomblog.nsf/dx/ocsp-stapling-and-why-it-matters-improve-your-https-performance.htm

The Domino TLS Cache supports OCSP caching and provides OCSP data to the HTTP task,
but the back-end code does not distinct between certificates with or without OCSP information.

The TLS Cache will re-check the status every 10 minutes. So it's just a small overhead in the back-end thread.
But it's still good to disable when your certificates have no OCSP responder available in the certificate.

Here is how to check if a certificate has OCSP responder.
Either check the full openssl -text output or just query the OCSP URI as shown below.

I have disabled OCSP stapling on my servers a while ago.

If you never enabled it on Domino, there is nothing to change.
But this information might be also important for other server type that might use OCSP in combination with Let's Encrypt.

openssl x509 -in buypass.pem -noout -text

Authority Information Access:
                OCSP - URI:http://ocsp.buypass.com
                CA Issuers - URI:http://crt.buypass.no/crt/BPClass2CA5.cer


openssl x509 -in buypass.pem -noout -ocsp_uri
http://ocsp.buypass.com

• Comments [0]

Domino IQ -- Paraphrasing explained

Domino IQ 14.5 GA introduces a paraphrasing request invoked before the actual request (added post 14.5 EA3).
This additional request is intended to protect against ticking the LLM to perform other operations then intended by the system prompt.
The idea is to send a request to the same LLM to sanitize and refine the request.

The functionality is documented here -> https://help.hcl-software.com/domino/14.5.0/admin/conf_security_considerations_for_iq.html

The additional request adds overhead and depending on the LLM model it might introduce undesired results.
My personal recommendation is to disable this additional check, because most LLMs today already provide protection and also contain guard functionality for additional protection.

To disable the functionality add the following notes.ini

DOMIQ_DISABLE_PROMPT_PARAPHRASE=1

In my simple example, the meaning of the query slightly changed ("sense" vs "purpose").
I have seen other requests where a German question got translated to English, which causes the response to be returned in English in some cases.

The red part is the system prompt and prefix in front of the actual query.
The actual request uses the configured system prompt with the query returned by the LLM from the first request.


Paraphrasing request is send to the same LLM

{"model":"Qwen2.5","messages":[{"role":"system","content":"Paraphrase the following text:"},{"role":"user","content":"Paraphrase the following text:What is the sense of life?"}],"max_tokens":1024,"temperature":0.0}


Actual request

{"model":"Qwen2.5","messages":[{"role":"system","content":"You are a knowledgeable and helpful assistant. Provide clear and accurate answers to user questions in a concise and friendly manner."},{"role":"user","content":"What is the purpose of life?"}],"max_tokens":1024,"temperature":0.0}

• Comments [0]

Domino 14.5 Design task comes with some pretty useful new functionality

First of all the design task is now leveraged for updating all system databases during template setup for a new release.
The only important detail to know is that the database needs to have the admin server set to that server.

The background is that an automated process should not run wild on all replicas in your domain trying to upgrade the design multiple times.

There is also a new list of system databases automatically maintained: dominosystemdbs.ind.
The file is updated when the server starts and is leveraged by the design task.
I blogged about it before, but now here it is live in action.

The design task now writes a separate log file IBM_TECHNICAL_SUPPORT/design_upgrade.log.
The file is automatically rotated like other system files with a date and cleaned up like other system logs when configured.

In this file you find additional information like if the admin server is set right or what got updated.
The file also contains run-time information per database and also information about the template version update.

In my case I had to run it again manually after server update, because my admin server was set to a different server for autoupdate.nsf.

Probably you don't want to look into the file on the server. Here is another option you might like.
You can run the design task on all system databases ( -sd ) and get them updated + get a mail with the log file ( -mt ).

If you want to use the standard design task operation only taking into account the admin server settings of the design task, just run the -sd switch.
In case you want the same behavior template setup uses, you can specify the new -adm option.
This would run the design task only on system database (or in general database) with the admin server set to this server also if the "Refresh on admin server only" option not set in the database.

In any case you would get warnings in design_upgrade.log if no admin server is set or "Refresh on admin server only" is not set.

Here is what I just did to upgrade my autoupdate.nsf design after I realized the admin server was wrong and I corrected it.

lo design -sd -mt "daniel nashed"

Refreshing design [autoupdate.nsf]
Updating 'ac Download' into database 'Domino Auto Update' from template 'Domino Auto Update'
Updating 'ac Edit' into database 'Domino Auto Update' from template 'Domino Auto Update'
Updating 'ac Enable' into database 'Domino Auto Update' from template 'Domino Auto Update'
Updating 'ac Formula Check' into database 'Domino Auto Update' from template 'Domino Auto Update'
...
Design refresh completed (1 sec) [autoupdate.nsf] 14.5 (02.01.2025 21:21:08) -> 14.5 (10.04.2025 16:57:53)


Mail example from production today




New Design Task Switches in Domino 14.5


-sd        Refresh design of System Databases in dominosystemdbs.ind.
-adm       Refresh design only if current server is Administration Server of database.
-mt name   Mail design update log to the Internet Address name.

• Comments [0]

Notes/Domino/Traveler 14.5 is available

Domino 14.5 is out and there is not much to say ...
... because if you are running Domino 14.0 and have Auto Notify enabled, you should see the ProductInfo updated already.

Inside the Notes, Domino and Traveler Product Info documents you find details + links to release notes, what's new and other resources.
See the screen shot below.

There are exciting new features specially on the Domino side.
Some of them will need some detailed info later and I am covering most of it in my sessions at DACHNUG conference next week.

Domino Container project update

I have also pushed out the Domino Container project and I have updated all my servers already.
The Windows machines use AutoUpdate. Most of the Linux machines use the container image.

Along with the Domino 14.5 update, I added the REST API 1.1.4 to the container build.

Adding software means not only changing the software.txt, but also running an end to end automation test after building an image.
This ensures, software is downloadable, installs and runs.
Most of this process is fully automated and is executed before any Git hub repository update -- not just for software updates, but also changes in the project.

To upgrade a container based server, you just build a new image. The build menu remembers the selected software when you save the selection and picks the latest versions automatically for you at build time.
Below I have an example how that looked for me today when updating one of the DNUG Lab servers including all the add-on software.






Domino container inspect example switching from 14.5 EA3 to the release + Updating the Domino REST API


Info: New Image Version available!


------------------------------------------------------------------------------------------
 Runtime        :  docker 27.5.1
 Compose        :  docker-compose 1.29.2, build unknown
 Status         :  running
 Health         :  healthy
 Started        :  04.06.2025 09:43:03

 Name           :  /domino
 Image          :  hclcom/domino:latest
 Version CNT    :  14.5EA3
 Version IMG    :  14.5
 Domino Ver CNT :  14.5EA3
 Domino Ver IMG :  14.5
 Add-ons CNT    :  ontime=11.1.1,verse=3.2.4,nomad=1.0.16,traveler=14.5EA3,domrestapi=1.1.3.1-14
 Add-ons IMG    :  ontime=2.3.0,verse=3.2.4,nomad=1.0.16,traveler=14.5,domrestapi=1.1.4-14
 Base Image     :  registry.access.redhat.com/ubi10/ubi-minimal
 BuildTime CNT  :  03.06.2025 16:28:46
 BuildTime IMG  :  17.06.2025 06:45:16

 Hostname       :  ray
 Volumes        :  /local/docker
 Mounts         :  /local

------------------------------------------------------------------------------------------
 Container ID   :  bbf67511762b
 Image-ID CNT   :  90e39ef92404
 Image-ID IMG   :  59f7ca155342
------------------------------------------------------------------------------------------

 Image Size     :  2180 MB
 Network Name   :  host
 Driver         :  overlay2
 Restart Policy :  always
 Restart Retries:  3

• Comments [0]